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ABSTRACT 


Student persistence in online learning environments has typically 
been studied at the macro-level (e.g., completion of an online 
course, number of academic terms completed, etc.). The current 
examines student persistence in an adaptive learning environment, 
ALEKS (Assessment and LEarning in Knowledge Spaces). 
Specifically, the study explores the relationship between students' 
academic achievement and their persistence during learning. By 
using archived data that included their math learning log data and 
performance on two standardized tests, we first explored student 
learning behavior patterns with regard to their persistence during 
learning. Clustering analysis identified three distinctive patterns of 
persistence-related learning behaviors: (1) High persistence and 
rare topic shifting; (2) Low persistence and frequent topic 
shifting; and (3) Moderate persistence and moderate topic 
shifting. We further explored the association between persistence 
and academic achievement. No significant differences were 
observed between academic achievement and the different 
learning patterns. We interpret this result in addition to a 
preliminary exploration of topic mastery trends, to suggest that 
"wheel-spinning" behaviors coexist with persistence, and is 
ultimately not beneficial to learning. 
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1. INTRODUCTION 


Assessment of LEarning in Knowledge Space (ALEKS) is an 
online adaptive learning system built based on Knowledge Space 
Theory [8]. According to Knowledge Space Theory, a knowledge 
domain is represented by a finite set of concepts. The knowledge 
state of a student in a domain can be represented by a particular 
subset of concepts that the student is capable of mastering. By 
gauging learner’s knowledge state, ALEKS determines what a 
student knows and is ready to learn, and provides personalized 
learning paths that are ideal for each student [3]. When a learner 
first use ALEKS, the system starts with an individualized initial 
assessment to find the student’s knowledge state. The assessment 
usually consists of 20 to 30 problems (out of more than 600 
problems). After the initial assessment, the student receives a 
report in a color-keyed pie chart (as shown in Fig. 1). Each "slice" 
of the pie chart corresponds to a particular area of the syllabus, 
and the darker shades of color indicating how much the student 
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has mastered in that area [1]. After the first assessment, ALEKS 
identifies the student’s knowledge state and generates a list of 
topics the student is ready to learn in each area. Once a student 
chooses the area and topic he/she wants to work on, ALEKS will 
provide a set of problems, and the student learns by solving 
problems under a specific topic. After successfully solving 
problems covering the same topic, the system will determine a 
student’s mastery of the topic and the add the topic to the 
student’s knowledge pie, and the student can then move onto a 
new topic [2]. 


} Algebra (46 of 71) » Whole Numbers (76 of 80) 


> Proportions, Percents, 
and Probability (13 of 33) 


> Fractions and Decimals (52 of 66) 
> Measurement and Graphs (32 of 48) 


> Geometry (36 of 72) 


Figure 1: ALEKS knowledge pie showing number of concepts 
learner has learned and needs to learn 


As one of the popular adaptive learning systems, ALEKS was 
evaluated in some empirical studies which were carried out in 
different settings, and was observed to be effective in most of the 
studies [6, 9, 12, 13, 16, 19]. These studies generally measured 
ALEKS students’ learning gains or academic achievements; 
however, none of them looked at students’ learning process, or 
online learning behaviors. In this study, we explored students’ 
offline learning outcomes and online learning behavior patterns, 
and investigated whether persistence was associated with 
academic achievement in an individualized online learning 
environment. We further examined students’ wheel-spinning 
behaviors [5] in order to understand the association. 


2. RELATED WORK 


In this section, we will introduce how persistence has been studied 
in different learning contexts--traditional classroom environment 
and online learning environment, and how the relationship 
between persistence and academic achievement has been 
investigated. Persistence is “the quality that allows someone to 
continue doing something or trying to do something even though 
it is difficult or opposed by other people” [15]. According to 
Rovai, persistence is the behavior of continuing action despite the 
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presence of obstacles [22]. Persistence in the face of adversity is 
often described as a result of high motivation. For instance, in the 
literature investigating classroom learning, persistence was 
typically examined as an outcome factor of motivation. Elliot and 
his colleagues [7] found mastery goals and performance approach 
goals were positive predictors of persistence; Vansteenkiste et al. 
[24] found intrinsic motivation improved student persistence; 
Multon et al. [18] proved that self-efficacy facilitated persistence. 
Although the concept of persistence was studied in different 
literature, it was operationalized in various ways. For example, in 
the meta-analysis by Multon and his colleagues [18], they 
summarized three ways of operationalizing persistence after 
viewing eighteen studies-- time spent on task, number of items or 
tasks attempted or completed, and number of academic terms 
completed. Apart from these three commonly used measures, 
persistence was also frequently measured with self-reports [4, 7, 
27). 


In the context of online learning environment, persistence was 
usually defined as the completion of an online course, or an 
antonym of attrition [10, 14, 20, 22]. Persistent learners, who were 
referred to as “completers”, were the learners who successfully 
completed an online course. Non-persistent learners, who were 
referred to “dropouts”, were the learners who did not finish a 
course [10, 14]. Persistence was mainly explored as a dependent 
variable affected by psychological and social factors, such as self- 
motivation, engagement, economic support, etc. [14]. Persistence 
was also investigated as a consequence correlated with online 
behaviors such as participation, discussion, etc. [17, 21]. 


Despite various studies on persistence in learning, persistence was 
rarely studied as a predictive factor. Stekel and Tobias [23] 
hypothesized a curvilinear relationship between self-estimated 
persistence and achievement. They predicted a moderate amount 
of persistence would lead to the highest achievement. They also 
hypothesized that persistence would be positively related to 
achievement in lecture-related instructional environment, but 
unrelated in the individualized instructional environment. 
However, they failed to prove their hypotheses. While examining 
the mediation effect of persistence on the relationship between 
goals and academic achievement, Elliot et al. [7] found self- 
reported persistence was a positive predictor of exam performance 
in lecture-based classroom setting. This proved one of Stekel and 
Tobias’ hypotheses. For online learning system like ALEKS, the 
instructional context could be considered individualized because 
ALEKS models student’s knowledge state and always provides 
the concepts students are ready to learn. Therefore, we wonder 
whether persistence is unrelated to academic achievement in the 
individualized learning environment like ALEKS. 


3. METHODS 
3.1 DATA SETS 


The data sets used for this study were collected from Jackson- 
Madison Intelligent Tutoring System Evaluation (JMITSE) 
program. JMITSE was an after-school program applied in five 
middle schools in Jackson-Madison County School System of 
Tennessee from 2009 to 2012. The goal of JMITSE program was 
to investigate whether technology outperformed human teachers 
in math teaching. There were two experimental conditions: 
teacher condition and technology condition. In the teacher 
condition, students learned math with math teachers in the after- 
school program. In the technology condition, students learned 
math with ALEKS. For this study, we only used data from the 


ALEKS condition. The program lasted for three academic years 
and 366 sixth-graders were assigned to the ALEKS condition 
altogether. Participants were supposed to study for two one-hour 
sessions every week, for twenty-five weeks. Logs of all students’ 
online learning activities were recorded by the system. The 
ALEKS log file included students’ online ID, the topics (i.e. 
concepts) students attempted, learning mode (ie. learning, 
review), time elapsed and the result of each attempt. For each 
attempt, there are five possible results: correct, wrong, explain, 
added to pie and failed. “Correct” is shown after a learner 
attempts a task and gets the correct answer. “Wrong” is shown 
after a learner attempts a task and gets a wrong answer. After a 
learner gets a wrong answer, two buttons “Try” and “Explain” 
will be shown to the learner. If the learner hits the “Try” button, 
he/she will be given another problem to work on. If the learner 
hits the “Explain” button, a worked example of that problem will 
be provided (as shown in Fig. 2). Reading an explanation is 
regarded as an attempt and the result is recorded as “Explain.” 
“Added to Pie” is shown after learner attempts a problem 
correctly. The difference between “Added to Pie” and “Correct” is 
that “Correct” is based on one single attempt, but “Added to Pie” 
is based on multiple correct attempts. When a learner can 
correctly answer problems under a concept consistently, ALEKS 
decides the learner has mastered the concept and adds the concept 
to the learner’s knowledge pie. After being added to the 
knowledge pie, that topic will not be given to the learner again, 
except for reviewing. “Failed” is shown after a learner attempts a 
task and answers incorrectly. Similar to “Added to Pie”, it is not 
merely based on one single attempt, instead, it happens when 
there are multiple unsuccessful attempts and the system decides 
that the learner failed to learn that topic. 


The participants of JMITSE took the Tennessee Comprehensive 
Assessment Program (TCAP), which is a standardized test, twice. 
Before entering the program, the students took TCAP5, which 
was TCAP for Sth graders. After finishing the program, the 
students took TCAP6, which was TCAP for 6th graders. The two 
tests were used as pretest and posttest in the analysis. 


3.2 DATA PROCESS 


The log file used in this study contains 366 students’ 330,319 
lines of online learning sequence. Each line represents an attempt 
from a student on one topic. Most students attempted multiple 
topics, and most topics were attempted multiple times. Therefore, 
for each student, there were multiple rows of data. Firstly, the data 
was aggregated at topic level. After aggregation, the number of 
observations for each individual student equaled to the number of 
topics they attempted. For each topic attempted by a student, we 
computed the number of attempts and amount of time spent on the 
topic, as well as whether it was mastered. We named the variables 
“Attempt”, “Time” and “Master’. Pearson product-moment 
correlation coefficient indicated that “Attempt” and “Time” were 
highly correlated (r=.98). To determine which variable to use as 
the measure of effort, we further examined the distribution of the 
two variables. The distribution of the two variables revealed that 
neither of them were normally distributed. However, after log 
transformation, “Attempts” became approximately normally 
distributed, but “Time” was still skewed (as shown in Fig. 2). 
Therefore, “Attempts” was chosen to measure student’s effort on 
task. Next, we created three variables as measures of persistence 
and dummy coded them. They were “High persistence”, 
“Moderate persistence” and “Switch”. While “High persistence” 
and “Moderate persistence” were used to describe different levels 
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of persistent learning behaviors, “Switch” was used to describe 
non-persistent behaviors when a student gave up a topic quickly 
and switched to a new topic before mastery. For a topic, if its log- 
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Figure 2: Distribution of log-transformed attempts and log- 
transformed time on each topic 


transformed attempts were in the fourth quartile of the 
distribution, “High persistence” was coded 1, otherwise it was 
coded 0. If its log-transformed attempts fell into the second or 
third quartile of the distribution, “Moderate persistence” was 
coded as 1, otherwise it was coded 0. For “Switch”, both attempts 
and the result were taken into account. If a topic’s log- 
transformed attempts was in the first quartile of the distribution, 
and the topic was not mastered, “switch” was coded 1, otherwise 
it was coded 0. After the new variables were created and coded, 
the 51,982 rows of data were aggregated to student level by 
averaging the persistence variables, and we got 366 observations. 
After second aggregation, the three persistence variables became 
continuous rather than binary. These variables represent the 
percentage of topics that a student persisted at each level. For 
instance, if a student gets 0.2 in “high persistence”, it means that 
the student attempted twenty percent of the topics with high 
persistence. Lastly, we computed the number of topics each 
student attempted for data screening. The three persistence 
variables were percentages, which represented the percentage of 
topics attempted with some type of behaviors. If the total number 
of topics attempted by the were too small, it did not necessarily 
imply certain behavior patterns, even if the percentage for that 
behavior was high. Therefore, we decided to screen the students 
who only attempted a small number of topics. Based on the 
distribution of topics attempted by each student, the students 
whose attempted number of topics fell within the first quantile 
(Topics<=61) were screened from further analysis. There were 
275 observations after screening. 


After data process, we conducted cluster analysis to explore 
students’ persistence learning patterns. We performed analysis of 
covariance to compare academic achievements of students from 
different groups to explore the association between online 
behavior and academic achievement. We also conducted analysis 
of variance to compare the mastery topics between groups to 
better understand the association. 


4. RESULTS 
4.1 CLUSTER ANALYSIS 


There is no strictly defined sample size for cluster analysis. 
According to the suggestion of Formann [11], the minimal sample 
size should be no less than 2 * cases (k = number of variables), 
preferably 5*2“. The study examined the clustering of 275 
observations across three variables, which fell comfortably within 
the accepted range. Ward’s [25] hierarchical clustering technique 
was applied and the squared Euclidean distance was used to 
calculate the distance between clusters. A scree plot was used to 
determine the optimum number of clusters, where the levelling- 
off point indicated a reduced variability between clusters after it 
[26]. Examination of scree plot revealed flattening between three 
and four clusters, indicating that a three-cluster solution best 
captured the similarities and differences between students on the 
three variables. The cluster membership did not change by 
repeating the analysis, and significant differences were found by 
conducting ANOVAs for the clustering variables, which further 
confirmed the quality of the solution. The three-cluster solution is 
shown in Fig. 3. The scales are the percentage of topics students 
attempted with a specific behavior. The scales are the percentage 
of topics students attempted with a specific behavior. For 
example, the y axis of the top row is the percentage of switch 
behavior. The x axis of the top middle block is the percentage of 
moderate persistent learning behavior, and x axis of the top right 
block is the percentage of high persistent learning behavior. From 
the top middle block, we can find the clusters are more distinct on 
switch behavior (i.e. y axis), whereas on the moderate persistence 
behavior (i.e. x axis) there is more overlap between the student 
clusters. From the top right block, we can find the black cluster 
has more high persistent learning behavior, and the green and red 
clusters have more overlap. The descriptive statistics on the 
grouping variables and the academic achievement variables, that 
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we further explored, are shown in Table 1. 


Figure 3: Scatterplot matrices of three-level persistence of 
three clusters 
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Cluster 1: High persistence, low switch 


Cluster 1 (i.e. the black cluster in Fig. 3) accounts for 37.5% of 
the study sample (n=103). The students in this cluster switched 
topics less than members of other two clusters. The switching 
ratio of cluster 1 is 0.16, which indicates that students quickly 
gave up or switched to other topics before mastery for 16% of the 
tasks they attempted. For 34% of the tasks, the students worked 
with moderate persistence (i.e. attempted the task for 3-7 times). 
And for 31% of the tasks, the students worked with high 
persistence (i.e. attempted the task for 8 or more times). These 
students did not easily give up on tasks, and put a large amount of 
effort on one third of the tasks they got, which indicated that they 
were persistent learners. 


Table 1: Mean scores and standard deviations for each 
variable by cluster 


Cluster 1 Cluster 2 Cluster 3 
(n= 103) (n= 54) (n= 118) 
Switch 0.16 0.36 0.23 (o=0.05) 
(o=0.05) (o=0.05) 
Moderate 0.34 0.28 (o=0 0.34 (o=0.05) 
persistence (o=0.05) .05) 
High 0.31 0.19 0.18 (o=0.04) 
persistence (o=0.07) (o=0.07) 
TCAPS5 46.72 39.37 47.28 
(o=18.25) (o=17.60) (o=17.23) 
TCAP6 43.23 32.69 40.49 
(o=20.89) (o=18.44) (o=21.63) 


Cluster 2: Low persistence, high switch 


Cluster 2 (i.e. the red cluster in Fig. 3) is a comparatively smaller 
cluster including 19.6% (n=54) of the study sample. The 
distinctive characteristics of this cluster is their high switching 
ratio. For 36% of the tasks they were given, the learners quickly 
gave up or switched to new tasks before mastering them. The 
students worked with moderate persistence (i.e. attempted the task 
for 3-7 times) on 28% of the tasks. And worked with high 
persistence for 19% of the tasks (i.e. attempted the task for 8 or 
more times). Compared with the other two clusters, the students in 
this cluster were not very persistent. Although they worked on 
some tasks with multiple attempts, they gave up on a large 
percentage of the tasks, and they were not willing to put too much 
effort on a task. 


Cluster 3: Moderate persistence, moderate switch 


Cluster 3 (i.e. the green cluster in Fig. 3) is the largest cluster with 
118 students representing 42.8% of the study sample. The student 
in this cluster switched topics on 23% of the tasks, which is higher 
than that of Cluster 1 but lower than that of Cluster 2. They 
worked with moderate persistence on 34% of the tasks and with 
high persistence on 18% of the tasks. Compared to the other two 
clusters, this cluster does not distinctively stand out in any type of 


behavior. The students gave up a medium portion of topics and 
worked with high effort on a comparatively low portion of topics. 
They worked on the tasks with mostly moderate persistence. It 
seems they were regulating their learning in a rational way in the 
self-regulated learning environment. 


4.2. ANALYSIS OF 
(ANCOVA) 


In order to investigate the association between persistence and 
academic performance, a one-way analysis of covariance 
(ANCOVA) was conducted to determine a statistically significant 
difference between three clusters on posttest scores controlling for 
pretest scores. The effect of cluster on posttest scores after 
controlling for pretest scores was not statistically significant, 
F(2,212) = 1.25, p = .29, which means the academic achievement 
of the three clusters with different behavior patterns were not 
significantly different from each other. 


COVARIANCE 


4.33 ANALYSIS OF VARIANCE (ANOVA) 
AND POST HOC TESTS 


In order to understand why persistence was not related to 
academic achievement, we further examined the percentage of 
topics attempted with moderate persistence and high persistence. 
For clusters one, two and three, the percentages of tasks attempted 
with moderate persistence without mastery were 0.11 (o = 0.05), 
0.08 (o = 0.04) and 0.07 (o = 0.03), respectively. The percentages 
of tasks attempted with high persistence without mastery were 
0.21(6 = 0.08), 0.17 (o = 0.06) and 0.16 (co = 0.06). Analysis of 
variance (ANOVA) indicated a significant difference of the 
unmastered topics attempted with moderate (F (2, 272) = 30.3, p< 
.001) and high persistence (F(2,272) = 14.3, p < .001) among the 
three clusters. Post-hoc tests indicated Cluster 1 was significantly 
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higher than both Cluster 2 and Cluster 3 in unmastered topics with 
both moderate and high persistence. This provides some insight as 
to why persistence did not make a difference in learning: the 
students were wheel-spinning [5]. We explored two highly 
attempted topics in our data sets and found the probability of 
mastering those topics got close to zero after a certain number of 
attempts (as shown in Fig. 4). This indicates the existence of 
wheel-spinning. 


Another one-way analysis of variance (ANOVA) was conducted 
to determine a statistically significant difference between three 
clusters on the number of mastered topics at different difficulty 
levels. The topics were divided into three levels based on the 
percentage of students who mastered them. The topics in the first 
quartile had the highest mastery percentage, which we defined as 
easy topics. The topics in the second and third quartiles had the 
medium mastery percentage, and were defined as medium topics. 
The topics in the fourth quartile, had the lowest mastery 
percentage, and were defined as hard topics. The numbers of 
mastered easy topics were not found to be significantly different 
among three clusters, F (2,272) = 2.56, p = .08. However, the 
numbers of mastered medium (F (2,272) = 9.98, p = 0) and hard 
topics (F (2,251) = 8.92, p = 0) were found to be significantly 
different between clusters. Post-hoc tests indicated that cluster one 
and three mastered significantly more medium and hard topics 
than cluster two, but there was no statistically significant 
difference between cluster one and three. The means and standard 
deviations of the number of topics mastered by each cluster are 
shown in table 2. 


Table 2: Means and standard deviations of the number of 
topics mastered by each cluster 


Cluster 1 Cluster 2 Cluster 3 
Easy topics 24.06 22.7 (o=12.88) 27.21 
(o=12.1) (o=15.08) 
Medium topics 49.46 33.56 51.75 
(o=26.85)  (o=18.74) (o=26.99) 
Hard topics 16.11 6.86 (o=6.05) 14.27 
(o=13.93) (o=12.13) 


5. DISCUSSION AND CONCLUSION 


In previous research, student persistence has only been measured 
by macro-level data (e.g., completion of an entire course). This 
study took a different approach by examining persistence at a 
more micro-level; specifically, we looked at student persistence 
within specific tasks in the ALEKS learning system. We were 
able to extract three distinct clusters of persistence related student 
behaviors through cluster analysis. The students in the high 
persistence cluster put medium to high effort in most of the topics 
they attempted, and they rarely switched to a new topic before 
mastery. The students in the moderate persistence cluster put 
medium effort in most topics they attempted and they did not 
easily give up topics before mastery. The students in the low 
persistence cluster frequently switched to new topics before 


mastery, often giving up tasks after one or two attempts. The 
comparison of students’ academic achievement in the three 
clusters did not reveal any significant difference. This result is 
consistent with the hypothesis proposed by Stekel and Tobias 
[23], who suggested that persistence and achievement are 
unrelated within individual learning contexts. Although learning 
gains were not different between clusters in standardized tests, the 
mastery of topics was found to be different. The more persistent 
clusters--cluster one and cluster three-- mastered more medium 
and hard topics than the non-persistent cluster--cluster two. This 
suggests persistence was associated with learning in ALEKS, 
especially for more difficult topics. The inconsistency between 
learning gain in ALEKS and TCAPs might be related to different 
topics covered in ALEKS and TCAPs. 


It is worth noting that the pretest and posttest assessments present 
a limitation to the current analysis. The TCAPS and TCAP6 were 
used as pretest and posttest measures, and may cover different 
concepts that are not well aligned. However, a further look at the 
possible reasons behind non-productive persistence suggested 
wheel-spinning might relate to ineffective learning. That is, even 
though students were persistently working on a single topic, they 
appeared to be at an impasse. These impasses were not resolved 
with more attempts, which ultimately resulted in the student never 
mastering the topic. Although ALEKS has a system that can 
detect ineffective learning and provide feedback, like “Failed”, to 
learners, the percentage of “Failed” was very low (1.e., 1%). In 
many cases, learners were struggling and wheel-spinning, but the 
system did not stop them with a “Failed” indicator, or any other 
type of intervention. Therefore, we suggest ALEKS to improve 
the mechanism to detect wheel-spinning and provide intervention 
in a timely manner. 
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